Skip to content

♻️ Extract Celery code to a new simcore library #7698

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 124 commits into from
Jun 23, 2025
Merged

♻️ Extract Celery code to a new simcore library #7698

merged 124 commits into from
Jun 23, 2025

Conversation

@giancarloromeo
Copy link
Contributor

@giancarloromeo giancarloromeo commented May 19, 2025 •
edited
Loading

What do these changes do?

This PR refactors Celery integration by extracting wrapper logic into a dedicated simcore-celery-library package and updates related CI/CD and dependency configuration.

  • Introduces a new simcore-celery-library package with packaging and setup files
  • Adds and configures requirements, Makefiles, and version metadata for the new library
  • Updates CI workflows, CodeQL, VS Code settings, CODEOWNERS, and Codecov to include the new package

No new features have been added, but a small redesign improves significantly the usability of the Celery interface. Any dependency with a specific App Server (i.e. FastAPI or aiohttp) has been removed and a BaseAppServer wrapper acts as a proxy, hiding the real technology used.

DISCLAIMER

A new queued_task package has been created in servicelib that contains protocols and models independent from Celery technology: they should stay there. A more suitable name can be decided in the future or - better - a general interface that include also old long running tasks.

Related issue/s

How to test

cd packages/celery-library
make install-dev
make test-dev

Dev-ops

Nothing

@giancarloromeo giancarloromeo added this to the Bazinga! milestone May 19, 2025
@giancarloromeo giancarloromeo self-assigned this May 19, 2025
@giancarloromeo giancarloromeo added the a:storage issue related to storage service label May 19, 2025
@codecov
Copy link

codecov bot commented May 19, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 60.28708% with 83 lines in your changes missing coverage. Please review.

Project coverage is 87.70%. Comparing base (4ee83ca) to head (eda052e).
Report is 1 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #7698      +/-   ##
==========================================
- Coverage   88.01%   87.70%   -0.32%     
==========================================
  Files        1846     1430     -416     
  Lines       71249    59396   -11853     
  Branches     1225      649     -576     
==========================================
- Hits        62709    52092   -10617     
+ Misses       8187     7085    -1102     
+ Partials      353      219     -134
Flag Coverage Δ
integrationtests 64.24% <ø> (+<0.01%) ⬆️
unittests 86.01% <60.28%> (-0.60%) ⬇️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library ∅ <ø> (∅)
pkg_celery_library 81.13% <88.31%> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library 71.56% <0.00%> (-1.06%) ⬇️
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 85.10% <ø> (+0.05%) ⬆️
agent 96.29% <ø> (ø)
api_server 92.31% <ø> (ø)
autoscaling 96.03% <ø> (ø)
catalog 92.29% <ø> (ø)
clusters_keeper 99.13% <ø> (ø)
dask_sidecar 91.79% <ø> (ø)
datcore_adapter 97.94% <ø> (ø)
director 76.73% <ø> (ø)
director_v2 91.17% <ø> (+0.01%) ⬆️
dynamic_scheduler 96.69% <ø> (ø)
dynamic_sidecar 90.09% <ø> (ø)
efs_guardian 89.65% <ø> (ø)
invitations 93.00% <ø> (ø)
payments 92.57% <ø> (ø)
resource_usage_tracker 89.00% <ø> (+0.21%) ⬆️
storage 86.31% <80.55%> (-1.23%) ⬇️
webclient ∅ <ø> (∅)
webserver 87.63% <ø> (-0.03%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ee83ca...eda052e. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@giancarloromeo giancarloromeo changed the title ♻️ Extract Celery code to a new package ♻️ Extract Celery code to a new simcore package May 19, 2025
@giancarloromeo giancarloromeo changed the title ♻️ Extract Celery code to a new simcore package ♻️ Extract Celery code to a new simcore package May 19, 2025
@giancarloromeo giancarloromeo changed the title ♻️ Extract Celery code to a new simcore package ♻️ Extract Celery code to a new simcore package ⚠️ May 20, 2025
GitHK
GitHK reviewed Jun 18, 2025
View reviewed changes
Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a few questions

bisgaard-itis
bisgaard-itis approved these changes Jun 18, 2025
View reviewed changes
Copy link
Contributor

@bisgaard-itis bisgaard-itis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool! Thanks a lot for the heroic effort on this one!

pcrespov
pcrespov reviewed Jun 18, 2025
View reviewed changes
@giancarloromeo giancarloromeo added 🤖-automerge marks PR as ready to be merged for Mergify and removed 🤖-do-not-merge (optional) blocks Mergify from merging the PR labels Jun 20, 2025
@giancarloromeo giancarloromeo enabled auto-merge (squash) June 20, 2025 15:33
@giancarloromeo giancarloromeo requested a review from sanderegg June 23, 2025 07:31
sanderegg
sanderegg approved these changes Jun 23, 2025
View reviewed changes
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pair reviewd

@sonarqubecloud
Copy link

sonarqubecloud bot commented Jun 23, 2025

Quality Gate Passed Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@giancarloromeo giancarloromeo merged commit 6bdc814 into ITISFoundation:master Jun 23, 2025
95 of 97 checks passed
@giancarloromeo giancarloromeo deleted the extract-celery-code branch June 23, 2025 20:03
@giancarloromeo giancarloromeo mentioned this pull request Jun 27, 2025
Merged
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Aug 5, 2025
Closed
88 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pcrespov pcrespov pcrespov approved these changes

Copilot code review Copilot Copilot left review comments

@GitHK GitHK GitHK approved these changes

@sanderegg sanderegg sanderegg approved these changes

@YuryHrytsuk YuryHrytsuk YuryHrytsuk approved these changes

@matusdrobuliak66 matusdrobuliak66 matusdrobuliak66 approved these changes

@bisgaard-itis bisgaard-itis bisgaard-itis approved these changes

@mrnicegyu11 mrnicegyu11 Awaiting requested review from mrnicegyu11

Assignees

@giancarloromeo giancarloromeo

@bisgaard-itis bisgaard-itis

Labels

🤖-automerge marks PR as ready to be merged for Mergify a:storage issue related to storage service

Projects

None yet

Milestone

Bazinga!

Development

Successfully merging this pull request may close these issues.

7 participants

@giancarloromeo @GitHK @pcrespov @sanderegg @YuryHrytsuk @matusdrobuliak66 @bisgaard-itis